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ABSTRACT 

Accountability in vocational education iic discussed 
as consisting of two subsystems, one providing data about developing 
new programs and the other as providing data for answering questions 
about the quality of existing programs. The Minnesota model for 
product evaluation, the latter subsystem, is discussed. It is 
predicated on the theory that vocational programs serve a dual 
role — they provide society with a source of skilled workers where 
there is a demand for skilled workers (social mai itenance) and they 
provide the individual with skills to progress in an occupation where 
his/her needs can be fulfilled. The Minnesota Satisfaction 
Questionnaire (to be completed by a student) and the Minnesota 
Satisfaction Scales (to be completed by the immediate supervisor) 
were chosen as measurement instruments for a pilot test of the model, 
and a study was conducted to determine the sensitivity and 
reliability of these scales before the pilot test. Ss were all the 
students who graduated from a poet-secondary vocational program in 
one of nine curriculum clusters in Minnesota in 1968-69 and who were 
employed one year after graduation, a total of 1,229. Results were 
not positive, but evidence for rejection of the scales was not 
conclusive. Suggestions for further research are made. (KM) 
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INTRODUCTION 

It has been said that the concept of accountability is much like taxes; 
It's difficult to cope with but a necessary part of the total system. In 
this day and age, when funds are limited and financial commitments are depen- 
dent upon the ability of educational programs to demonstrate their effective- 
ness and efficiency, accountability assumes an even more important role than 
it has in the past. 

The Minnesota Research Coordinating Unit for Vocational Education has - 
been concerned with the problem of accountability for vocational education 
find is conducting programs of research aimed at developing and implementing 
such a system in the Stite. The purposes of this presentation are to share 
with you some of i.he id'.:as which have been generated as well as some of the 
findings our resectrch hdS produced. This presentation is divided into the 
following three parts: (1) A brief description of our concept of account a- 
.bility, (2) a dip^iussiou of a model for "produ,at-.evaluationIl,^ aud.jCS).. a. — 
summary of the results if research to test the sensitivity of selected criterion 
measures for prc^nct evaluation. • 

-ACCOJNTABILITY — — - ' 

r ■ ' ' 

Educational accountability is a construct which suggests that vocational 
educators are re:v)ons:.ble for offering high quality programs to all students 



who v;ant and can benefit from such programs. Conceptually, it may be viewed 
as a total system for collecting information concerning program objectives 
such that educators can make decisions to improve both the quantity and 
quality of their programs. In this sense, a total system of accountability 
is perceived as a tool for decision-making v/hich requires a systematic pro- 
cedure for colJecting, analyzing, and interpreting empirical data about 
programs. 

In^terms of the kinds of decisions which vocational educators are typi- 
cally required to make and the data requirements for making these decisions, 
a total system of accountability must necessarily consist of two separate but 
related data collection subsystems. 

The first subsystem deals with decisions about the quantity, type and 
location of programs. This is most often referred to as a subsystem for 
program planning and development which attempts to answer such questions as: 
Are programs satisfying the demand for vocational training? Are programs 
serving the numbers and kinds of students it should be serving? What new 
programs should be offered and where should they be located? What programs 
should be termina:ed as a result of insufficient demand? In order to answer 
these kinds of question 5, a data collection system which attempts to obtain 
an adequate match betwem labor supply and demand is necessary.^ 

The second t^pe of subsystem deals with decisions about the quality of 
existing vocational pre ;rams and is typically referred to as "program evalua^ 
tion". The concern here is to obtain information about how "good" or how 
"successful" a p^Dgram is and then make decisions with respect to improving 
or terminating programs. In most instances, program operators have relied 

^The Minnesota RC(J ^s currently developing and testing a model for program 
planning that will attempt to (a) identify the needs of students K to adult, 
and (b) match these needs with a demand for skilled labor and vocational 
training programs. 



upon information or data only about the quality of the instructional ^'process". 
In comparison, relatively little has been done to develop an evaluation sub- 
system that attempts to assess the quality of programs in terms of the 
on-the-job behaviors of former students. 

In summary, a total system of accountability is perceived as a process 
of collecting and analyzing empirical evidence to make decisions about either 
the quantity or quality of existing vocational prograis. A total system, 
therefore, consists of two subsystems: One siibsystem provides data about 
developing new programs (program planning) and the other provides data for 
answering questions about the quality of existing programs (program evaluation). 

PRODUCT EVALUATION: AN ACCOUNTABILITY SUBSYSTEM 

"~ 'The Minnesota RCU has long been concerned with the problem of developing 
a state operated evaluation subsystem. The State of Minnesota has already 
made advancements in developing an evaluation procedure which centers on 
assfessing the quality of the instructional process. Both self evaluation 
techniques and evaluation by independent teams composed of educators and 
representatives from industry have been used to assess the quality of each 
vocational curriculum ii the State. Because there is also a need to assess 
the quality of vocation \1 programs In terms of their products, the Minnesota 
RCU has been givGi the responsibility to continue developing a rationale and 
model for evaluating, tl t quality of post-secondary vocational programs in the 
State. The rema-lidf^r of this report is devoted to a discussion of the model 
for product evaluation and a summary of the results of research that has 
been conducted. 

Purpose of Vocational Education 

The model for product evaluation is predicated on the principle that 
public vocational education programs are developed to enhance the mutual 



satisfaction of both societjr and the individual. That is, vocational programs 
serve a dual role: they provide society with a source of skilled workers 
who are capable of performing satisfactorily in an occupation where there is 
a demand for skilled workers (social mainter^ance) and they provide the 
individual with skills to obtain and make progress in an occupation where his/ 
her needs can be fulfilled (self-fulf illment) . 

Figure 1 illustrates the purposes of public vocational education programs 
and the relationship among these purposes. Since vocational education is 
primarily concerned with the on-the-job performance of former students, its 
major thrust is towards improving the work adjustment of the individual, and 
maintaining an appropriate distribution of skilled workers in the labor force. 
Each of these purposes is circumscribed by the concept that vocational educa- 
tion is responsible for providing training opportunities to all students who 
want and can benefit from such training. 

The concept of work adjustment has been defined as the correspondence 
between (a) the abilities of the individual and the ability requirements of 
the job, and (b) the needs of the individual and the need reinforcers (satis- 
fiers) of the job. To the extent that correspondence is obtained, an 
Individual is said to have achieved "work adjustment" which will lead to a job 
or occupational tenure. 

For the purpose of product evaluation, work adjustment and the maintenance 
of an appropriate distribution of workers in light of the labor demands repre- 
sent the criterion for assessing program effectiveness. The distribution 
of workers in terms of labor demand is determined by whether a former student 



'See V. Dawis, L. N. Lofquist and D. J. Weiss A Theory of Work Adjustment 
(a revision), Minnesota Studies in Vocational Rehabilitation , xiii, 
Minneapolis., University of Minnesota, 1978. 
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obtained employment ir a job unrelated to training, directly related to 
training, cr broadly related to training. 

In order to pilot test and further develop the model, two instruments 
which purport to measure the construct of work adjustment were selected be- 
cause of their potential for use as criterion measures for product evaluation* 
Students complete the Minnesota Satisfaction Questionnaire (MSQ) and his/her 
immediate supervisor completes the Minnesota Satisfaction Scales (MSS). 
Work satisfaction (hov; satisfied an individual Is with his job) is measured 
by the MSQ which consists of three scales: (1) Extrinsic Satisfaction, 
(2) Intrinsic Satisfaction, and (3) General Satisfaction. Satisfactory job 
performance is measured by the MSS and consists of five scales: J(l) Perfor- 
mance, (2) Conformance, (3) Dependability, (A) Personal Adjustment, and (5) 
General Satisfactoriness. 

Program evaluation which attempts to assess the "quality" of the program 
is dependent upon comparing the actual^ observed outcome of a program with 
some standard. There are three types of performance standards which may be 
used: (1) an arbitrarily fixed standard, (2) standards based on norm groups, 
or (3) comparative standards. Since there was no available rationale for 
establishing a fixed standard for either satisfaction or satisfactoriness 
and since norms have no: been developed on these measures, the comparative 
standard was selected the basis for evaluating vocational programs. This 
means that only prograr s which provide training for the same work roles are 
to be compared in termrs of their relative effectiveness in the improvement 
of the work adju tment of the individual. 

A production function, a concept borrowed from economics and used in 
•conjunction with regression techniques, 1'^ to be used to make comparisons 

%or a more complete explanation of job^-relatedness see Wheeler, D. N. 
Technical report no> 5: The measurement of job relatedness for vocational 
program evaluation . Minneapolis: Minnesota Research Coordinating Unit for 

• Vocational Education, 1971. 



among programs within each vocational cluster while simultaneously controlling 
for differences in student characteristics, socio-economic conditions, and 
other intervening variables. Since the validity and sensitivity of the two 
measures of work adjustment to serve as criterion measures for the evaluation 
of vocational programs has not been established, this phase of the evaluation 
model has not been completed. 

In summary, the model for product evaluation is based on the principle 
that the purpose for public vocational education is to improve the work 
adjustment of individuals and to maintain an appropriate distribution of 
skilled workers in the labor force such that the mutual satisfaction of both 
the individual and society are maximized. The criterion measures selected 
for comparing vocational programs,^ using a production function, regressional 
analysis technique, were the Work Adjustment Projects' measures of the individ- 
uals' work satisfaction and satisfactory job performance. 

RESEARCH CONCERNING CRITERION MEASURES FOR PRODUCT EVALUATION 

One of the major problems involved in developing a state operated pro- 
duct evaluation subsystem is the lack of sufficient data concerning the 
characteristics of former students and their subsequent on-the-job performance. 
Fortunately, the State of Minnesota has been able to develop a comprehensive 
data base consisting of measures on all Minnesota post-secondary vocational 
school students (about 20,000) for a two year period as a result of a feder- 
ally funded project called Project MINI-SCORE and a State financed Vocational 
Follow-up Project. With the cooperation of Drs. Howard F. Nelson and David J. 

Pucel, it was possible for the Minnesota RCU to begin testing the previously 

f 

discussed model for product evaluation. * 

The majpr purpose of our research effort was to determine the sensitivity 
and reliability of the Work Adjustment Projects' measures of job satisfaction 
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and sa:isfactoriness which are to be used as criterion measures for pilot 
testing the evaluation model. In order to examine the validity of these instru- 
ments for our purposes, answers to the following four questions were sought: 

1. Are measures of satisfaction and satisfactoriness sufficiently 
sensiti^e to discriminate among programs within a curriculum 
cluster? 

2. Are programs within a curriculum cluster homogeneous with respect 
to the types and/or characteristics of the students enrolled? 

3. Can variations in measures of satisfaction and satisfactoriness 

be explained (accounted for) by measures of student characteristics? 

4. Are the measures of job satisfaction, and satisfactoriness reliable? 
Population 

The population consisted of all the students who graduated from a post- 
secondary vocational program in one of nine curriculum clusters in Minnesota 
during the year 1968-1969 and who were employed one year after graduation. 
Table 1 lists the nine curriculum clusters and the number of programs and 
students within each cluster. Curriculum clusters were selected to insure 
representation according to sex* Only schools which had a total of ten or 
more employed graduates during the two year period were selected for analysis. 
In all cases, comparisons were made only between programs within curriculum 
clusters. Analyses between clusters have already been completed by the staff 
of Project MINI -SCORE. 

TABLE 1 



SUMMARY OF THE POPULATION ACCORDING TO CURRICULUM aUSTER, AND 
NUMBER OF SCHOOLS AND STUDENTS WITHIN EACH CURRICULU14 CLUSTER 
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Curriculum 


No. of Programs 


No. of Students 


Power and Home Mechanics 


5 


60 


Practical Nursing 


11 


299 


Auto Mechanics 


13 


129 


Mechanical Drafting and Design 


7 


70 


Machine Shop 


4 


59 


Welding 


8 


56 


Cosmotology 


6 


91 


Accounting 


12 


165 


Clerical 


14 


300 


TOTAL: 


80 


1229 



/ 
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Findings 

Question //I: Are measures of satisfaction and satis factoriness sufficiently 
sensitive to discriminate among prt^grams within a curriculum 
cluster? 

One-way analysis of variance (i^NOVA) was used to compare programs with- 
ing each of the ni* e curriculum clusters on the basis of the mean scale scores 
for both the three scales of job satisfaction (MSQ) and the five scales of 
job satis factoriness (MSS). Data v/ere computed using responses of students 
who were classified in each of the following three categories of job related- 
ness: (1) employed, (2) employed in a job directly related to their training, 
(3) employed in a job broadly related to their training. 

No significant differences (.05 level) were found with any of the 

scales for either the MSS or MSQ at each of the three levels of job relatedness. 

This finding can be explained in one of three ways: (1) the instruments may 

not be sensitive to differences among programs, (2) variation in the measures 

might be due to error of measurement (unreliability), or (3) programs could 

be, in fact, homogeneous and therefore differences are not expected. Subse- 

quent analyses were coi ducted to determine the viability of these alternatives. 

Question ffli Are progrims within a curriculum cluster homogeneous with 

respect t> the tyjpes and/or characteristics of the students 
enr»>lled? 

One-way ANOVV was ised to compare programs within each of the nine 
curriculum clusters on ^he basis of their mean scale scores on each of the 
following instruments: General Aptitude Test Battery (GATE), 16 Personality 
Factors (16PF), Minnesota Vocational Interest Inventory (MVII), Minnesota 
Scholastic Aptitule Te; t (MSAT), Vocational Development Inventory (VDI) and 
the Minnesota Imp^rtanne Questionnaire (MIQ). The scales for each of these 
instruments are listed, in Appendix A. Analyses were conducted on the scores 
of students who were classified according to three previously mentioned 
levels of job relatec'ness. 

O 
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With only minor exceptions (certain scales of interest, personality and 
occupational needs), no significant differences were revealed. This strongly 
suggests that program operators tend co have similar entrance requirements 
and tend to enroll students who have quite similar characteristics. 

This finding also suggests that one of the reasons that these measures 
of job satisfaction and satii^f actoriness could not detect differences among 
programs was because there was no differential affect of programs on students 
who were relative.^y similar when they entered the program. Subsequent 
analyses were necessary to determine whether the variat3.on in the two cri- 
terion measures can be accounted for by student characteristics or error of 
measurement. 

Question it3: Can variations in measures of satisfaction and satisf actoriness 

be explained (accounted for) by measures of student characteristics? 

Multiple correlation v/as used to assess the amount of variation that 
the GATB and MVII could account for in each scale of the MSQ and MSS for 
only those students employed in a job broadly related to the training program 
from which they had graduates. In this instance, only schools having at 
least 30 graduates were used for the analysis. This assured an adequate 
ratio of the number of variables to the number of students. In few 
Instances were the resulting multiple correlation coefficients statistically 
significant at the .05 level. This finding suggests that student character- 
istics, at least in terms of measured aptitude an I interest, are not related 
to the actual on-the-job performance of graduate as measured by job 
srtisfaction-satisf actoriness measures in question. This finding may be 
explained in one cf three ways: (1) student characteristics may be unrelated 
to performance measures, (2) perhaps student characteristics that are related 
to these performance measures have not been measured or, (3) the ratio of 
true variance to error variance is so sm;ill that large correlations are not 




expected. 
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Question #4: Are the measures of job satisfaction and satisf actoriness 
reliable? 

Previous research conducted by the Work Adjustment Project at the 
University of Minnesota revealed that the scales of both the MSS and MSQ 
have satisf actoril/ high reliability coefficients ranging from about .65 to 
~»-98.- Only a minimal amount of research has been conducfp-^ ^ 'he test-retest 
stability of the instruments dealing with population o. .dents in a psy- 
chology class and older, employed workers. These results indicate that the 
Instruments do produce stable results (especially over short periods of time). 
However, since none of these research efforts dealt with a population similar 
to the age level of post-secondary vocational school graduates, additional 
research concerning the reliability of these instrume.nts seemed warranted. 

Table 2 shows the coefficients of internal consistency for each scale 
of the MSQ and MSS for three heterogeneous groups of students who graduated 
from the post-secondary vocational system in Minnesota. 

TABLE 2 

COEFFICIENTS OF INTERNAL CONSISTENCY FOR TEE SCALES OF 

THE MSQ AND MSS 



MSQ 



Group 1 



Group 2 



Group 3 



Intrinsic ,870 

Extrinsic ,783 

General Satisfaction ,896 

MSS 

Performance ,900 

Conformance , 841 

Dependability ,887 

Personal Adjustment .738 

General Satisf actoriness .946 



.885 
.881 
.915 

.911 
.873 

.906 
.821 

.954 



.871 
.800 
.896 

.680* 

.837 

.848 
.706 

.925 
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Table 2 shows coefficients of internal consistency that are within a 
satisfactory range (.680 to .954). This finding leads one to believe r.hat 
the variation on the scale scores of both the MSS and MSQ are not due solely 
to erro : * asurement, but may, be due to "true" variation in on-the-job 
performance. Until additional research pertaining to the test-retest 
stability of the instruments is conducted, a final assessment of the instru- 
ments cannot be made. 

CONCLUSIONS 

The results of research concerning the utility of the MSS and MSQ as 
criterion measures for a state-operated sub-system of product evaluation 
have not been as positive as we would have liked. However, evidence for 
their rejection is not conclusive. Also, the results of research reported 
here do not detract from or invalidate the rationale for the model < r product 
evaluation presented. The concept of work adjustment and maintaining an 
appropriate distribution of skilled worke*"s still remain as acceptable pur- 
poses for public school vocational education programs. The findings do 
suggest, however, that additional research is needed in order to perfect a 
state operated subsystei of product evaluation. The following implications 
for further research ari suggested: 

1. Addition: al research concerning possible sources of unreliability 
of the MSS anc MSQ should be undertaken. 

2. Differe t stai.dards of program comparison (fixed standards or 
normative bases) should be developed and tested. 

3. Measure* of job satisfaction and satisf actoriness should be ob- 
tained . loser to the point of graduation in order to detect any 
differences vhich may be due to the educational program. 

A. If post-secondary vocational programs are found to be equally 
effective producers of satisfactory and satisfied graduates, 
then the relative efficiency of these programs should be 
compared. 
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5. Graduates of other forms of vocational training (private, MDTA, 
apprenticeship, or cooperative programs) and graduates of 
post-secondar>' institutions might be compared. 

6. Means of assessing job satisfaction and satisf actoriness other 
than those developed by the VJork Adjustment Project may be 
developed and tested. 
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APPENDIX A 



Instruments and Their Respective Scales Used 
to Estimate Student Characteristics 



Minnesota Vocational Interest Inventory - (1) Mechanical, (2) Health 
Se- (3) Office Work, (4) Elect ronics, (5) Food Service, (6) Car— 
pentry, (7) Sales-Office, (8) Clean Hands, (9) Outdoors. 

General Aptitu de T est Bat tery - (1) G - intelligence, (2) V - verbal 
aptitude, (3) i: - numerical aptitude, (4) S - spatial aptitude, 
(5) P - form perception, (6) Q - clerical perc;iption, (7) K - motor 
coordination. 

16 Personality Factors - (1) A - reserved, (2) B - less intelligent, 
(3) C - emotional, (4) E - humble, (5) F - sober, (6) G - expedient, 
(7) H ~ shy, (8) I - tough minded, (9) L - trusting, (10) M - practical, 
(11) N - forthright, (12) 0 - placid, (13) Ql - conservative, (14) Q2 - 
group-tied, (15) Q3 - casual, fl6) Q4 - relaxed. 

Minnesota Scholastic Aptitude Test - only one score 

Vocational Development L.ventory only one score 

Minnesota Importance Questionnaire - (1) ability utilization, 
(2) achievement, (3) activity, (4) advancement, (5) authority, (6) company 
practices and policies, (7) compensation I, (8) co-workers, (9) creativity, 
(10) independence, (11) moral values, (12) recognition, (13) respon- 
sibility, (14) security, (15) social service, (16) social status, 
(17) supervision (human relations), (18) supervision (technical), 
(19) variety, (20) working conditions, (21) work challenge, (22) com- 
pany image, (23) organization control, (24) feedback, (25) physical 
facilities, (26) work relevance, (27) company prestige^ (28) company 
goals, (29) closure, (30) compensation II. 
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